Skip to content

fix(gdpval): count a model that delivers nothing as a lost battle, not a masked row - #6077

Merged
AdamRajfer merged 3 commits into
NVIDIA-NeMo:mainfrom
AdamRajfer:arajfer/gdpval-no-deliverable-loss
Nov 16, 2026
Merged

AdamRajfer merged 3 commits into
NVIDIA-NeMo:mainfrom
AdamRajfer:arajfer/gdpval-no-deliverable-loss

Conversation

@AdamRajfer

@AdamRajfer AdamRajfer commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

When the scored model delivers nothing (it runs out of wall time, or stops on its own context or turn limit), GDPVal masked the row: it left the denominator and the pairwise Elo fit, so a model scored higher the more often it failed. This is a GDPVal scoring change.

  • Comparison mode (gdpval): an agent_timeout row (stop_reason=wall_time) is now a success row holding a battle the model lost against every active reference (num_comparison_trials losses per reference repeat, judge_response.manual_imputation=model_failure_as_loss). It stays in the denominator and in the Elo fit, with failure_kind=agent_timeout and failure_outcome_attribution=model. The same holds when the run stopped on context_limit, max_iterations or max_turns and there is no deliverable (model_context_limit / agent_limit, failure_outcome_action=count_as_zero). The existing multistage coverage gate no longer blocks a stage because of such a row.
  • Rubric mode and aa_briefcase_lite (which share resolve_deliverables; it has no Elo fit): an agent_timeout is an unmasked zero with the same kind and attribution.
  • Unchanged: judge errors and judge-side faults (judge_failed, permanent judge errors, transport ineligibility), missing reference files, a lost session, and a missing deliverable with no model stop reason keep their masked/terminal handling and retry or regrade; none is ever scored as a loss. agent_timeout carries no outcome action, so recovery still reruns a timeout whose window held a recorded server-side fault and treats a clean one as the model's final result.

Validation

  • pytest resources_servers/gdpval/tests resources_servers/aa_briefcase_lite/tests: 1076 passed, 1 skipped. Without the app change the new and updated tests fail (7 failures).
  • New tests: an agent_timeout is a lost battle with no judge call; context and turn limits are counted losses; a missing deliverable without a model stop stays a terminal failure; a no-deliverable row lowers the win rate (0.75 to 0.5625 in the fixture) and the Elo versus the same run with the row masked; the multistage strict stage now completes with a timeout row and plans the next stage; the aa_briefcase_lite timeout is an unmasked zero.
  • pre-commit run --from-ref origin/main --to-ref HEAD passes.

Rollouts

N/A: unit tests over verify and aggregate_metrics; no live run.

Compatibility and benchmark impact

Scores go down for models that time out or fail: each such task now counts as a loss (comparison mode) or a zero (rubric, AA-Briefcase-Lite) instead of being dropped, so slow or failing models score lower, and the drop grows with how often they fail. Earlier results that masked these rows are not comparable with new ones. The comparison/imputed_loss_* metrics now include these rows.

Checklist

  • I have read the contributing guidelines.
  • The change is focused.
  • Tests added or updated and pass locally.
  • Documentation: N/A, no documented behavior of this path.
  • Pre-commit checks pass locally.
  • New source files include the required Apache-2.0 SPDX header: N/A, no new files.
  • All commits have DCO sign-off.

…t a masked row

When the scored model ran out of wall time (or stopped on its own context or
turn limit) with no deliverable, GDPVal masked the row as agent_timeout: it left
the denominator and the pairwise Elo fit, so a slow or failing model scored
higher the more often it failed.

In comparison mode such a row is now a success row that holds a battle the
model lost against every active reference, with its failure_kind and model
attribution; the fit keeps it. In rubric mode and in aa_briefcase_lite it is an
unmasked zero with the same kind. An agent timeout carries no outcome action,
so recovery still reruns one whose window held a server-side fault. Judge
failures, transport and missing-reference faults are unchanged.

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team November 12, 2026 19:13
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HOLD — the new timeout rows can bias multistage scoring, and AA’s reported rates still exclude them.

I reviewed all five changed files and traced their affected verification, aggregation, persistence, recovery, and cross-stage reuse paths. The inline findings describe the correctness issues. This was static inspection; I didn’t execute tests or independently verify the reported test results.

RISK — Required benchmark validation is missing

The PR explicitly reports no live rollout or reward-profiling run. This changes scoring for the fixed GDPVal benchmark and its AA consumer. The pinned rubric requires a before/after reward comparison, and the repository quality bar requires representative real model rollouts for behavior-changing environment work. The synthetic win-rate fixture establishes arithmetic, not the end-to-end reward distribution.

Could you add representative rollout artifacts and before/after numbers using the same model, tasks, references, judge settings, and budgets? Include GDPVal win rate/Elo, scalar rewards, model-stop frequencies, and masked/scored counts; include AA’s reported rates for the shared timeout path. The expected change is confined to the intended model failures: ordinary delivered results and judge/reference faults must retain their handling. Acceptance is linked run evidence showing that separation and explaining the observed score shift. No production-code patch is needed for this evidence request.

NOTE — Update the documented stage-completion rule

The GDPVal benchmark guide still says an answered timeout holds no battle and makes a strict stage fail. That now contradicts the changed verifier and multistage test. Could you replace that paragraph with the clean-timeout counted-loss rule and note that results using the earlier masking policy aren’t comparable? Preserve the separate explanations for infrastructure failures, missing references, and partial-completion policies. Static acceptance: the guide should agree with the clean-timeout regression test and no longer recommend a partial-completion policy merely to accommodate such a row.

Note: formal approval is not available for this request, so this review is posted as a comment only.


Inline comments (escaped v1):

  • resources_servers/gdpval/multistage_orchestrator.py line 267: BLOCKER — Infrastructure timeouts now enter calibration as model losses.

    Removing _ng_failure_class from GDPVal’s timeout response makes _is_success_row accept it even when the shared is_infrastructure_agent_timeout classifier returns true. A concrete trigger is a wall_time result whose last captured model call returned HTTP 503. The retry predicate then sees no failure class, so its imputed losses feed calibration instead of being retried. Resume selection also gates the persisted success permanently. Before this change the same timeout carried class agent_timeout and remained retryable.

    Single-pass collection already handles this success shape in RolloutRecords._payload, but multistage bypasses that selector. Could you normalize infrastructure-attributed timeout results into excluded, retryable failures before routing, retry selection, and pooling? Apply equivalent classification when loading saved rows, reopening dependent stages through the existing restart machinery. The invariant is that an infrastructure-cut execution window contributes no model-loss evidence. This spans several paths, so a one-line predicate change isn’t sufficient.

    Preserve clean timeouts as counted losses, the attempt cap and recorded reference assignment, and judge-only recovery without regenerating its saved answer.

    Proposed acceptance: Extend the real-verifier multistage fixture with a final captured call carrying response_metadata.status_code=503. With retries enabled, exclude that attempt’s losses and use the subsequent successful attempt. With retries disabled, persist and resume it; the affected stage must reopen. Retain the clean-timeout assertions: one attempt, counted losses, and the next stage planned. These checks weren’t executed.

  • resources_servers/gdpval/multistage_orchestrator.py line 1619: BLOCKER — A later stage regenerates a clean timeout instead of retaining its loss.

    The new imputed-loss success reaches this loop and is recorded as a produced deliverable, although a wall-time stop persists no files. If the task/repeat recurs with default reuse_cached_deliverables=True, build_stage_rows marks it for reuse. The real cached resolver returns 404, and run_rollouts runs the policy again. A successful second generation can therefore replace the final model failure in the headline stage.

    The changed test misses this: its fake runner recognizes task index 0, while later stages offset that index, and it never exercises cached verification’s 404 fallback.

    Could you retain final no-deliverable model outcomes separately from reusable file artifacts, keyed by task and repeat, and re-impute their losses against later stages’ assigned references without generating again? Cover timeout/context/turn-limit outcomes when cross-stage reuse is enabled, including file-backed resume. Preserve ordinary cached-file judging, the fallback for a genuinely missing artifact with no saved final model outcome, and explicit reuse_cached_deliverables=False. The cache representation needs a deliberate choice, so I haven’t proposed a small patch.

    Proposed acceptance: Exercise two overlapping stages through run_e2e_multistage and the real cached resolver, with three ordinary results and one clean timeout in stage 0. Make the policy mock succeed if called again for that task. Assert one generation for that task/repeat and losses in both stages against their respective references, including after resume. Retain the existing missing-cache fallback for the distinct case without a recorded final model failure. Tests weren’t executed.

  • resources_servers/aa_briefcase_lite/app.py line 924: RISK — AA’s reported rates still exclude the newly scored timeout.

    Changing _AGENT_TIMEOUT to an unmasked zero makes this early return a scored row, but AABriefcaseLiteVerifyResponse defaults rubric_total and every pairwise count to zero. AA aggregation computes its rates from those counts, not reward. With the existing two-binary-check/two-reference fixture, one fully passing row plus one wall_time row still reports both rates as 1.0, rather than 0.5. These are the rates users are told to report, so the scalar zero doesn’t fix the reported benchmark scores.

    Could the unmasked timeout branch produce applicable failed-check and lost-trial counts, consistently with AA’s existing no-submission behavior? Preserve the timeout kind, attribution, and zero reward without judging partial files. Keep binary-only runs free of pairwise rates, and retain skipped-reference/excluded-judge rules and masked evaluation faults. Calling _grade(body) unchanged would grade partial files, so it isn’t a complete correction; the helper extraction needs author verification. The invariant is that each scored model zero contributes to every applicable reported rate’s denominator.

    Proposed acceptance: Aggregate a fully passing all row and the new timeout fixture; expect binary counts 2/4, pairwise counts 4 wins, 4 losses, both rates 0.5, and no judge call for the timeout. Retain the existing no-submission case, binary-only absence of the pairwise rate, and masked missing-directory/reference faults. Static reasoning only; tests weren’t executed.

  • resources_servers/gdpval/app.py line 1369: RISK — Live context/turn-limit runs with no files miss the new loss branch.

    A seeded run that never calls finish still gets a persisted directory containing finish_params.json = null and reference inputs: collect_deliverables always writes that marker, as the session test confirms. task_attempted() therefore returns true even though no deliverable exists. For max_turns, max_iterations, or context_limit, _model_stop_failure is never reached and the empty submission goes to the judge. If the judge is unavailable, this becomes a masked judge failure instead of the intended counted model loss. The new tests only use a nonexistent eval directory, bypassing normal session collection.

    Could you distinguish an attempted directory from one containing agent-produced files? A small replacement for the opening gate is:

    model_stop = _model_stop_failure(body)
    if (
        eval_task_dir is None
        or not task_attempted(str(eval_task_dir))
        or (
            model_stop is not None
            and not any(is_deliverable(path) for path in eval_task_dir.iterdir())
        )
    ):

    Keep the existing branch body, but remove its later duplicate model_stop = ... assignment. This uses the shared deliverable classifier and the collector’s flat output layout. Verify the suggestion before adopting it.

    Preserve judging when a stopped run has actual submitted files. Also preserve ordinary null/{} finish-marker judging without a model-stop reason, explicit stage-one missing-eval imputation, missing-reference handling, and the lost-session guard.

    Proposed acceptance: Parameterize the three stop reasons through a live seeded comparison session with no finish call; make any judge call fail the test and expect typed, unmasked losses. Repeat with a submitted report.md and expect normal judging. Retain test_existing_finish_marker_is_judged_with_loss_policy for its no-stop-reason cases. Tests weren’t executed.

  • resources_servers/gdpval/app.py line 665: RISK — A timeout becomes scored even when judging is explicitly disabled.

    With judge_deliverables=False and response.metadata={"stop_reason":"wall_time"}, this return bypasses the judging-disabled branch. Previously _AGENT_TIMEOUT supplied mask_sample=True; now GDPVal rubric and AA return an unmasked zero, while GDPVal comparison constructs counted losses. This breaks the masked, collection-only contract: a run with scoring disabled can acquire scored zero rows solely because it timed out.

    Could the timeout return preserve masking when judging is off? The smallest correction here is:

            return body, {"reward": 0.0, "mask_sample": not server.config.judge_deliverables, **_AGENT_TIMEOUT}
    

    This retains the existing timeout behavior of not collecting or judging partial work, keeps judging-enabled timeouts unmasked, and prevents the comparison caller from manufacturing losses in collection-only mode. It doesn’t change the pre-existing timeout persistence policy. Verify the suggestion before adopting it.

    Proposed acceptance: Parameterize GDPVal rubric/comparison and AA with judging enabled/disabled and wall_time. Enabled must retain the new zero/loss behavior; disabled must be masked, call no judge, and contain no imputed comparison loss. Retain the ordinary judging-disabled test’s persisted-file and gdpval:judging_disabled assertions, plus the timeout session test’s close/cleanup assertions. Tests weren’t executed.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

@yaoyu-33
yaoyu-33 requested a review from init-nikhil November 12, 2026 20:03
…ow writes

GDPVal no longer writes the _ng_failure_class=agent_timeout failure: a model
that ran out of wall time is a scored loss that keeps failure_kind=agent_timeout.
The legacy timeout signal registry named the removed line, so its producer check
failed. The entry now names the failure_kind form GDPVal writes, so health still
counts those rows as timed out. test_run_timing covers the same two forms.

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
@AdamRajfer

Copy link
Copy Markdown
Contributor Author

/ok to test 83226be

@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team November 12, 2026 21:00
@yaoyu-33
yaoyu-33 requested a review from pthombre November 12, 2026 21:02
@github-actions github-actions Bot added the sla:review-overdue Review response is over the one-business-day SLA label Nov 13, 2026
@AdamRajfer
AdamRajfer requested a review from agronskiy November 16, 2026 13:08
@AdamRajfer
AdamRajfer added this pull request to the merge queue Nov 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43890914423

@AdamRajfer
AdamRajfer removed this pull request from the merge queue due to a manual request Nov 16, 2026
@AdamRajfer
AdamRajfer added this pull request to the merge queue Nov 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43902912773

Merged via the queue into NVIDIA-NeMo:main with commit c4b7407 Nov 16, 2026
71 checks passed
@AdamRajfer
AdamRajfer deleted the arajfer/gdpval-no-deliverable-loss branch November 16, 2026 16:16
@svcnvidia-nemo-ci svcnvidia-nemo-ci added bug Something isn't working area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers ready-to-merge PR is approved, current, and waiting for final CI or merge labels Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers bug Something isn't working ready-to-merge PR is approved, current, and waiting for final CI or merge sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants